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ABSTRACT 

A research study investigated the validity of 
directly interfacing an uninitiated user with a complex, computerized 
batch processing system via a conversational, interactive language. A 
control group conducted mediated searches of the Educational 
Resources Information Center's (ERIC) files by consulting vith an 
information specialist. Members of the experimental group directly 
conducted their own searches, after undergoing training with the 
ERIC/QUERY Interface Program (EQUIP) , a computer-:ir:::sisted 
instructional training package. Results showed that: 1) the choice of 
interface methodology (mediated or direct) did not influence the 
user*s satisfaction with the search; 2) there was no relation between 
the user's knowledge of the system and his satisfaction with it; 3) 
those trained by EQUIP showed significant increases in their ability 
to conduct computer searches of ERIC; and 4) there was no predictive 
relationship between the user's knoTfledge of the system and the 
precision of the resulting search, thus indicating that the system 
was less than ideal. (PB) 



ERIC 



U.S. DEPARTMENT OF HEALTH. 
SOUCATION & WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 
THIS DOCUMENT HAS BEEN HEPHO 
DUCED EXACTLV AS RECEIVED PHDM 
THE PERSDM OR ORGANIZATION ORIGIN 
ATING IT POINTS OF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRE 
SENT OFFICIAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLICY 



A STUDY OF A DIRECT INTERFACE OF THE NOVICE 
USER TO A COMPLEX BATCH PROCESSED COMPUTER 
APPLICATION 

Albert D. Link 

South Carolina State Dept. of Education 



INTRODUCTION 

All too often, whether the average educator likes it or not, he 
is literally at the mercy of computer programmers, system analysis and 
the other "data processing types" in his quest to use the digital computer 
as a viable tool. It was not until the recent acceleration of the state- 
of-the-art of the computer industry that a "layman" educator could pro- 
ductively use the computer without the imposition of the expected inter- 
face of either learning programming or employing a "computer type" to 
do it for him. One such advance is the interactive (conversational) ser- 
vice that is rapidly becoming available within a feasible price structure. 

The conversational services under consideration is not unlike the IBM, 
supported A Programming Language (APL) which has enjoyed Increasing success 
on a national scope. What eire not under consideration are specialized 
on-line packages that have a single purpose or application. Such special- 
ized applications are still completely out of reach of the education 
community's budget. 

The prime objective of the study reported in this paper was to 
investigate the validity of directly interfacing an uninitiated user 
("layman" educator) with a complex, computerized batch processing system 
via a conversational, interactive language. 

The Scope of the Study 

Information is being produced by al * disciplines at an increasing 
rate each year. In response to this phenomenon, the United States Office 
of Education (USOE) has committed millions of dollars of the nation's 
resources to the creation of the first national education information 
system, known as "ERIC" (Educational Resources Information Center), to 
assist the decision-maker, researcher, and practitioner assess a growing 
bank of documents for the education profession. 

To aid the serious user in his document search and retrieval tasks, 
a computerized searching system known as QUERY was created and Implemented 
in over a dozen installations across the nation in 1969. QUERY is a 
batch processing system which requires a unique "search language" as 
^ machine-readable input and offers almost unlimited document searching 
Cr- capabilities. Because of this distinctive characteristic, if one is to 
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properly use the QUERY automated information system he must be able to 
select the correct search criterion prior to submitting his search request 
to the computer. 

Until the accomplishment of this study, all ''QUERY" installations 
accommodate the information seeker (user) by negotiating a search request. 
This normally required a co'nference between the user and an information 
specialist and was concluded when an acceptable search strategy (search 
language) based upon the user's expressed needs was recognized by the 
information specialist. This method of interfacing the system (through 
the information specialist) is less than satisfactory for a number of 
reasons including: (a) the difficulty the user has in expressing his 
specific search need to another person; (b) the size of the files often 
"overpowers" the unitiated user of the ERIC system; and (q) the highly 
technical search process cannot be completely explained to the user, thus 
his knowledge of the retrieval capabilities is severely limited^ 

The Objectives 

The objectives of the study were: 

(1) To develop, implement, and test a computer-assisted-instruction 
(CAl) package to train users of the computerized ERIC files to interface 
with the information retrieval system without the assistance of information 
or data processing specialists. 

(2) To design an instrument to measure the system user's satisfaction 
with the results of computer searches resulting from either the specialists 
or from personal interface with an information retrieval system. 

(3) To determine any differences between experimental and control 
groups of ERIC automated information retrieval system users — with the 
experimental group undergoing CAI training and direct job submission and 
the control group receiving no such training. 

The Hypotheses 

The specific hypotheses tested included: 

(1) There are no significant differences between the measured user 
satisfaction with computer search results of CAI-interfaced subjects 
and those undergoing search negotiations with an information specialist. 

(2) There are no significant differences between the subjects' 
pre- and post measures of knowledge of the information retrieval system 
administered during the experimental CAI treatment. 

(3) There is no significant relationship between measured knowledge of 
the ERIC automated information retrieval system acquired as a result of 
CAI training and user satisfaction as measured by the user satisfaction 
instrument. 
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The Basic Assumptions of the Study 



The study was based on the following assumptions: 

(1) Once an educator understands the utility of an information base 
especially designed and maintained for his discipline, he will attempt to 
capitalize upon the respective benefits of such a system. 

(2) Potential user traits and skills will vary extensively; reading 
ability, typing ability, and educational level (i.e. educators vs» trainees) 
will differ. 

(3) The ERIC information file size will not remain static; file size 
currently increases over one thousand documents per month (file size was held 
constant for the study). 

(4) For the duration of the study, computer files, search programs, and 
listing formats will be unchanged. 

(5) Measurement instruments are valid. 

(6) Subjects have had no prior experience with ERIC automated retrieval 
systems. 

Additionally, the QUERY information retrieval system, to be discussed later, 
is assumed to exist in a stable form. 

A BRIEF REVIEW OF RELEVANT LITERATURE 

General Systems Theory 

General systems theory as described by Optner (10, p. 9) and Stufflebeam 
(12, p. 124) is concerned with two major characteristics ~ input to the 
system and output from the system. When such systems involve computer usage 
they "... are structured, or designed, to operate in non variant, highly 
predictable ways Opener (10, P. 4)." It follows that if a highly structured 
system is not understood by its users, the efficacy of such a system will 
be less than design expectations. 

This study attempted to show that, v/ithin the realm of information 
retrieval systems, the less knowledge the user of the system possesses 
about the system, the greater the chances the system will not operate 
within its designed purpose. 

Decision Theory 

It is worthy to note that the various decision-making theories of 
Kepner and Tregoe (7, p. 73), Griffiths (4, p. 90), Halpin (5, p. 35), 
and Meyer (9, vol. 1) each rely on the acquisition of information as 
a prelude to completing the decision-making process. Additionally, each 
process is described as a system or may be thought of in terms of general 
systems theory. Stufflebeam (12, p. 38) points out that the availability 
of information prior to selecting alternatives in decision-making is 
assumed. Thus, the importance of information within the context of educational 
decision-making may not be overlooked. 
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The Use of Information Retrieval Systems 

The use of an automated retrieval system is anything but a simple 
matter. Lancaster (9, pp. 181-2) suggests that retrospective literature 
searches may be divided into those conducted without an intermediary by 
the person having the information need, and those delegated by this person 
to a second individual, usually a librarian or information specialist. 
Lancaster (8, pp. 182-5) clearly shows that there are different skills 
and expectations required of the requester under these two schemes. 
Additionally, he points out that twenty percent of the National Library 
of Medicine's Medical Literature Analysis and Retrieval System's searches 
involving defective interaction (between the user and the searcher) were 
judged to be of the type in which the requester, using an intermediary, 
was unable to precissly define his need except through some browsing in the 
literature (8, p. 184). To summarize, this means that to be successful 
the requester must spend a considerable amount of effort defining and 
negotiating his search request with the person who will actually encode the 
search in machine-readable form or learn how to use the system himself . 
The economics of the former appear to be less attractive than those of the 
latter. 

Batch Processing Automated Information Retrieval Systems 

Janda (6, p, 4) shows that in the fourteen years since tho first 
puDlished application to key word indexing of precoordinated information 
files by the computer, hundreds of batch processing information retrieval 
systems have been created. 

The review of the literature shows that many batch processing information 
retrieval systems were either designed for highly specific applications and/or 
developed around available computer hardware and software. 

The apparent advantages of the batch processing systems incliKje: 

(1) Systems designers may capitalize on their existing computer capa- 
bilities without added costs for additional software* 

(2) Maximal specialization of the system may be facilitated, allowing 
for "tailor-made" systems which meet the designer's needs at the least cost. 

(3) "In-house" created computer programs are maintained internally; thus 
changes to the system may be madj with the fewest constraints. 

Because of the advantages listed, it is unlikely that batch processing 
information retrieval systems (or many other typesi) will be discontinued 
without some significant changes in the costs and "state of the art" of the 
computer industry. 

It is noteworthy that, of all of the batch processing information 

retrieval systems investigated, none were accompanied with a system-to-user 

interface. All required the intervention of an intermediary, as cited by 
Lancaster (8, pp. 182-5). 



The ERIC/ QUERY Information Retrieval System 



The Educational Resources Information Center (ERIC) retrieval system. 
The ERIC system was begun in 1965 with the production and dissemination 
of the Catalog of Selected Documents on the. Disadvantaged , and since has 
grown into an international educationaT information retrieval system of 
considerable scope and complexity (1, pp. 58-63). Currently, ERIC files 
may be searched manually or via the computer using an on-line system called 
DIALOG or batch processing programs such as the USOE-sponsored QUERY system. 

DIALOG is a highly specialized, on-line system and is not under consi- 
deration due to its impractical cost. 

QUERY is an advanced, generalized, sequential, file-searching, batch 
processing computer software package, using RIE and CUE magnetic tapes 
or disk files, which is capable of isolating and listing any informational 
subset, depending on the search strategy used (3). 

CAI Studies 

Although the literature abounds with studies, projects, and programs 
which use CAI as a training media, no CAI programs or studies are known 
to exist which specifically focus upon the problem of requester-to-system 
interface. 

THE ERIC/QUERY INTERFACE PROGRAM (EQUIP) 

Available literature clearly showed the existence of numerous batch 
processing information retrieval systems as well as an increasing number 
of on-line interactive systems. The investment of resources for batch 
processing systems will most likely insure their existence and maintenance 
for years to come. The on-line systems are gaining some popularity despite 
their cost, due in part to the ability to provide systems-to-users interface, 
a feature totally lacking with the more numerous, less costly batch processing 
systems. This phenomenon led to the present study. 

Since no search interface package was known to vexist, the author created 
EQUIP (ERIC/QUERY Interface Program), a CAI training and search submisrion 
package henceforth described. 

The Program Language of EQUIP 

IBM*s popular telecommunication language, API, was selected as the 
language to program EQUIP because of its immediate availability, versatility, 
and power as a CAI medium. The version of API used was not standard, as 
supported by IBM, but instead has a unique feature provided through the 
efforts of Thomas H. Puckett of the New Mexico State University Computer 
Center. Dr. Puckett 's modification of API allows for the submission of the 
searches encoded by the user directly to the job stream of the IBM 360/65 
multiprocessing computer job stream — a feature not currently available 
in the IBM supported version of the language. 

In addition, API has been found to be efficient as related to computer 
time used for interface purposes. This characteristic coupled with some 
unique matrix operations and generic time-monitoring functions led to the 
decision to use API, a pre-dedicated service, as the CAI program "language 
for EQUIP. 



The Objectives of EQUIP 



The second and most important consideration relating to the creation 
of EQUIP was to determine a sound training foundation and minimum instructional 
objectives. This process relied heavily on the author's two years of 
experience while maintaining the QUERY system and processing automated 
searches for hundreds of users. Thus the central focus used in deriving 
minimal performance objectives for EQUIP are centered on the criteria: 
What is the minimum a novice user must learn for the implementation of a 
satisfactory computer search? Based on experiences with the system, the 
following guidelines were used. 

(1) The user must understand the proper use and meaning of three 
standard reference sources the ERIC Thesaurus of Descriptors , the ERIC 
Rotated Thesaurus of Descriptors , and the ERIC Posting of Descri ptor Stati sti cs 
(a source of the number of times a particular descriptor ic used by RIE) . 

(2) The user must be able to create a statement of his information 
need and subsequently reduce that statement to basic elemental descriptors. 

(3) The user must be able to properly use descriptors to isolate the 
citations of interest from the ERIC information files. 

(4) The user must understand and be able to correctly use the basic 
Boolean logic required in the QUERY search language. 

(5) The user must be able to correctly encode a search in machine- 
readable language. 

Since it was desired that the user-to-system interface program be suitable 
to individual differences (via CAI), other less important objectives were 
evident: 

1) The user must learn how to use the telecommunication terminal. 

2) The use^ must be aware of what the computer (QUERY) can and 
cannot do. 

The Development of EQUIP 

Based on the previously defined criteria, EQUIP was planned and written. 
Each subprogram was written and tested using both knowledgeable and novice 
subjects until the specific objectives of the subprograms were rewritten 
several times before they were deemed acceptable. 

After subprograms and related support programs were written, tested, 
and accepted, the total package was assembled and tested using knowledgeable 
and novice users of ERIC/QUERY facilities. Necessary modifications were 
installed and then "polished" until EQUIP was considered ready for formal 
testing and comparison with existing search negotiations procedures. 

An integral part of EQUIP is an eighteen-item pre- and post-test 
administered to the user in the course of his CAI training and subsequent 
search submission. The prime purpose for the pretest was that of switching 
(branching) the training sequence of the user with EQUIP, based on his 
knowledge of the ERIC system. It was assumed from the beginning that the 
skills and knowledge of users would vary widely. Through the use of the 
pretest switching provision of EQUIP, the training a particular user might 



undergo was, essentially, individually prescribed based on what ''he brings 
to the training session." It should be noted that the only major branching 
with EQUIP takes place as a result of the user's performance on the pre-test. 

Accordingly, four major "blocks" of instruction* as presented below, 
were conceptualized. 

Block one: system familiarization . The major purpose of this instructional 
block is to allow the user to learn the basic use of the APL telecommunication 
medium and some very basic concepts unique to computer systems. Internal 
subblocks and objectives are shown in the following table: 



Table 1 



Subblocks and Objectives of Block 1 of flQUIP 



Subblocks 



Objectives 



Terminal Training 



The user inust demonstrate competency In using the 
IBM 2741 telecomznunlcatlon terminal* 



Keyboard Training 



The user must demonstrate competency In using 
specific APL keyboard characters as well as 
knowledge of basic keyboard functions such as 
correcting mistake • 



Response Training The user must show competency in responding in 

the correct manner to instructions^ questions or 
commands used within EQUIP. 



Pretest 



A means of ascertaining the user's knowledge base 
relative to the QUERY automated Information retrieval 
system must be provided* 



Systems Training 



The user must demonstrate a basic knowledge of 
simple computer input, output and processing 
concepts. 



Computer Input 
Fami 1 lar Iza t ion 



The user is shown how the general search language 
(encoded) appears as input to QUERY* 



Computer Output 
Famil lar Iza t ion 



The user is invited to investigate sample listings 
of citations provided near the terminal* 



Note* — With the exception of the last two subblocks, the user must 
demonstrate competencies prior to being advanced further in the session* 



Block two: QUERY familiarization . The second major instructional 
block concentrates on the standard published materials deemed necessary 
to assist the user in his search construction. Table 2 depicts the 
subblocks and objectives of the second instructional block. 



Table 2 

Subblocks and Objectives of Block 2 of EQUIP 



Subblocks 



Objectives 



File Familiarization 



Rotated Thesaurus 
Familiarization 



Descriptor Statistics 
Usage 



Thesaurus Usage and 
Familiarization 



The user must show acceptable knowledge of 
the use of existing ERIC files and the- 
relationship and usage of descriptors in 
Isolating documents from a file* 

The user must Indicate that he has learned 
the use of the ERIC rotated thesaurus and 
its application to modified descriptors* 

The user must demonstrate knowledge and 
use of the ERIC descriptor reports, a 
resource material which aides in predicts 
ing the number of possible search hits* 

The user must demonstrate the ability to 
use descriptors, as given in the ERIC 
thesaurus, in relation to his information 
need statement* 



Note* Due to the wide variability of possible correct user 

recponses, progress Is monitored through the use of answers given to 
specific questions which are used to switch (or skip over) certain 
Instructional material* Progress is not always performance -based as 
was noted in the first block of instruction* 
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Bloc k three : sea» ch strategies. Only after the user has received 
appropriate training or has shown that he is knowledgeable of the concepts 
and materials offered in the first two blocks may he embark on block three, 
the most critical portion of the instructional sequence. Block three 
deals exclusively with the construction of the user's particular search 
strategy of interest, Subblocks and respective objectives are given in the 
following table: 



Table 3 

Subblocks and Objectives of Block 3 of EQUIP 



Subblocks 



Objectives 



Selecting Descriptors 



Logic Training 



Estimating Hits 
Famll iar izat ion 



Using previously learned concepts, the 
laser tentatively selects the descriptors 
he actctjilly intends to use for his search 
strategy. 

Using preselected descriptors from tht; 
previous subblock, all possible allowed 
I logic sequences must be presented and 
'understood by the user. It is at this 
point in the instruction that the user 
must indicate his understanding of the 
Boolean logic usp.d by QUERY. 

The user must be cognizant of the 
possible list his set of descriptors will 
produce, given all possible Boolean logic 
operator combinations. If the user is not 
satisfied, he must have the option of 
either changing his descriptors or selecting 
a different logic pattern. 
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Block four; practice and submission . The fourth and final block 
of instruction provides the user unlimited practice in creating his actual 
search, encoding it, and, subsequently, submitting it as a job to the computer 
job stream. Additionally, a posttest is administered, scored, and recorded 
at the end of this block. Table 4 presents the subblocks and instructional 
objectives of block 4. 

Table 4 

Su'-blocks and Objectives of Block 4 of EQUIP 



Subblocks Objectives 



Encoding the Search 



Diagnostics Rework and 
Submission 



Posttest 



The user must be given the opportunity to 
be actively involved in the encoding of 
his search. 

Mechanical encoding errors must be detected 
and feedback given to the user as he learns 
to encode his search. The us-^r must 
demonstrate competency in encoding his 
specific search of interest. The user's 
search request must be rejected by EQUIP 
if it fails the search language crlter-fA 
of QUERY. If the strategy is acceptable, 
the user must be allowed to submit his 
search directly to the computer job stream. 

A means of ascertaining the user's newly 
acquired knowledge of automated informa- 
tion retrieval via QUERY and EQUIP must 
be accomplished. 



The Instructional Sequence of EQUIP 

EQUIP utilizes a flexible instruction sequence based on initial 
switching as a result of user responses to the eighteen pretest items provided 
in block one. A simplified flowchart of the sequenced decision system is 
presented in Figure 1. It should be noted that not all switching of the^ 
sequence is based on pretest responses. Additional switching is accomplished 
via binary choice questions requiring honest user responses. For example, 
if asked "Do you know how to use the rotated thesaurus?" the user must 
respond "No" to receive needed training. Work space limitations prevented 
total competency-based testing of the user's knowledge — thus the "question 
and response" technique of controlling instruction sequence was necessary 
at times. 
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Figure 1 

Major Instructional Sequence of EQUIP 
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Figure 1 (ContdO 
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DESIGN OF THE STUDY 



The rationale of this study reduces to the following experimental 
question: After a CAI interface package has been created, is there any 
empirical evidence that its use with persons desiring information from 
the ERIC information base is effective? To phrase this question in another 
way: Is EQUIP as good as interface technique as existing search negotiation 
sessions with an information specialist? 

Subjects desiring to use the ERIC/QUERY information retrieval system 
were divided into two groups control and experimental. The experimental 
treatments are graphically portrayed in Figure 2. 

Both the experimental and control groups underwent a sequence of 
steps, shown as blocks in Figure 2, leading to the retrieval of a. specific 
set of surrogated documents (citations) which were intended to meet individual 
information needs. As depicted in Figure 2, the control group sequenced 
(top activities) differed from that of the experimental group (bottom 
activities). Activities common to both groups are shown in the center of 
the paradigm. The activities which differ, search negotiations versus 
CAI training via EQUIP, were those of pivotal interest to the study. 
(See Fiqure 2 next page) 

Sampling Design 

The study population . The population from which the study sample was 
drawn consisted of that group of people who have a need for information in 
the educational realm and are researchers, practitioners, or in training for 
the same. Additional population characteristics were: (a) male or female 
adults, (b) multi-ethnic, (c) must not have used ERIC/QUERY system previously, 
(d) must have been accommodated by the ERIC/QUERY facility, (e) must have 
been willing to spend the appropriate amount of time necessary to accomplish 
a search in person, and (f) must have been aware of ERIC/QUERY facilities. 
There is no reason to believe that the population described here would be 
different from persons seeking similar information in other geographic 
locations of the nation. 

Sampling technique . Since it was not possible to predefine a segment 
of the treatment population in terms of specific individuals, a modification 
of the systematic sampling method suggested by Sax (11, pp. 140-1), 
described in the subsequent paragraph, was used. 

A total sample size of sixty, thirty in each treatment group, was deemed 
necessary. Prior to conducting the experiment, a sequential randomized 
sign-in sheet was created using the random number generator feature of APL. 
Persons coming to ERIC/CRESS^ were required to sign in on the sheet described 
above. In this matter each subject was assigned to a treatment group at 
random. 



The ERIC clearing house, designated as ERIC/CRESS, at New Mexico State 
University was the site of the study. 
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Measures Employed 



Because of the nature of the search negotiations process (the control 
treatment) it was impossible to administ a pre- or posttest to members 
of the control group to determine any gain of knowledge of the ERIC/QUERY 
system. Since EQUIP was designed to impart knowledge gain, pre- and 
posttesting was confined to subjects in the experimental group. By this 
procedure, it was possible to determine if EQUIP was a successful CAI tool. 

To determine if any difference in user satisfaction of search effort 
was attributable to the experimental treatment, a second needed measure 
was accomplished at the termination of the treatment through the use of 
a user satisfaction questionnaire. 

The Pre-Post Measure 

It will be noted in the instructional sequence provided in Figure 1 
that the pretest was administered to members of the experimental treatment 
group immediately after competency in the use of the terminal has been 
demonstrated by the subjects. Similarly, the posttest, which was identical 
to the pretest, was administered after the subject completes his training 
session on EQUIP. 

Using Kuder-Richardson Formula 20, in the manner suggested by Sax (11, 
p. 161), the final session of the pre- and posttest used with the experimental 
treatment group had a reliability of .73, It should be noted that the 
EQUIP pre- and posttest instruments are "power*' and not *'speed" tests. 

The user satisfaction instrument . The experimental design employed by 
this study required that a comparative measure of treatment groups be 
administered. Again, due to the nature of the study problem, it was 
impossible to administer a pretest observation to both groups. If a 
subject had not been served there would be no value to measuring his 
satisfaction with the service, since no service would have been received. 
Thus, there was a need for only a post-observational measurement to deter- 
mine if the treatments used fostered any differences in satisfaction between 
treatment populations. 

It was rationalized tht when a user arrives at ERIC/CRESS, he has 
several expectations related to his specific informtional need: 

(1) He desires at least some hits as a result of his effort. 

(2) He wants document citations which are relevant to his information need. 

(3) He expects to expend time in his search effort. 

(4) He expects ERIC/CRESS to assist in his search effort. 

(5) He expects his total information research effort to be interesting, 
challenging, and useful. 

The extent to which the user's information retrieval experience fostered 
positive and/or negative reactions within each of the criteria] characteristics 
was measured with the simple Likert-type questionnaire. The questionnaire was 
created and tested with a small sample of subjects who had just received the 
results of an ERIC/QUERY computer search. 
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Sample characteristics . For the reasons discussed under sampling 
technique^ until the collection of data was completed no specific description 
of the study sample could be provided. Only after the experiment was completed 
and subsequent descriptive analysis applied could the sample be more clearly 
identified. 

There was a total of 31 men and 27 women in the sample; 16 men and 14 
women in the control group, 15 men and 13 women in the experimental group. 

AN OVERVIEW OF THE ANALYSIS 

Comparison of User Satisfaction Item Means 

Treatment group scores obtained from the five Likert-type scales of 
the computer retrieval evaluation form were compared using the t_ 
statistic in the manner suggested by Winer (13, pp. 31-33} for comparing 
uncorrelated means of two groups with unequal n's. To test the assumption 
of homogeneity of variance for each measure, the appropriate F ratios 
were computed as suggested by Winer (13, pp. 33-6). The critical value for 
the two-tailed t statistic when a = .05 and df = 56 (df = N. + Nk - 2) is 
t_;975(56) = 2.0T, a value obtained from Winer's tables (13, p. 6^1). 
The F ratio critical value is F qc(28,30) = 1.87 from the Chemical Rubber 
Company (CRC) Tables (2, p.308)"^^ 

It was noted that the assumption of equal variance of group scores for 
item three lacked support. This necessitated additional analysis of this 
single measure. Accordingly, the computations for testing differences 
between independent sample means (13, p. 37) with unequal variances were 
applied to data collected for item three of the user satisfaction 
questionnaire. A summary of these calculations is given in Figure 3. 



Figure 3 

Summary Calculations of t^' Statistic for Item Three 
of the Satisfaction Questionnaire 



Treatments 



N 

Means 
Variance 
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Control 
30 
3.7 
.286 



t'obs - 3.188135 
c « .28735 
f =45.53 
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Experimental 
28 
3.11 
.692 
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The values £ and f are based on Welch's (13, p. 37) derivation and 
approximation of the Student's t, distribution through a correction to the 
degrees of freedom which are used to ascertain the critical t^ value. Thus 
the critical value of t^ using the appropriate conversion, becomes t qjc (A) 
= 1975(46) 2.01 for a two-tail test. '^^^ 

It can be seen that the computed t value was outside the 1 075 critical 
range. Thus it is concluded that there were differences manifested between 
the treatment populations as related to the amount of time spent obtaining 
search results. 

Peformance of the Experimental Group 

Pre- and posttest measures . To provide evidence that the CAI 
training and search submission program (EQUIP) did increase the knowledoe 
of the user, the procedure suggested by Winer (13, pp. 39-43) for testing 
hypothesis between two means with correlated observations assuming a linear 
additive model was used. Calculations resulting in the values presented 
in Figure 4 were accomplished. 



Figure 4 

Summary of Calculations Comparing the Pre- and 
Posttest Interface Knowledge Gains of the 
Experimental Treatment Group 





Hypothesis Tested 


^ost 




^ost - ^pre 


^bs 




79.786 


34.071 


45.714 


14.09 



Since t_^95(28) = 2,05, t^bg, HojA. v^,^is rejected and it is concluded that 
the difference of 45.714 points average gain from pre- and posttest measures 
could not have occurred by chance at a = .05. 

Pre- and Posttest Scores Related to Satisfaction Score 

The possibility of a simple linear regressive association between the 
pre- and posttest gains made by subjects of the experimental group and 
their respective total scores on the satisfaction questionnaire was accomplished. 
If a predictive relationship could be found to exist, additional support for 
the treatment effect on user satisfaction would be gained. Using the APL 
linear regression program MREG provided by the NMSU computer center which 
assumes the linear model Y. - XB !.» regression analysis procedures were 
accomplished. A summary of these efforts is presented 1n Figure 5. 
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Figure 6 



Scattergram of Experimental Group Total Satisfaction 
Scores Versus Pre- and Posttest Gains 
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The results presented in Figure 5 agree with the scattergrams of the 
data. By inspecting the plotted values of pre- and posttest gains versus 
total satisfaction score. Figure 6, it is fairly obvious why the applied 
linear regression model accounts for only about 20 percent of the total 
variation'-- very little if any pattern exists. 

The previous discussion does not explain the significant regression 
manifested by the critical F value of the firiit calculation presented in 
Figure 5. By careful inspection of Figure 6 (scattergram of gains versus 
total satisfaction scores), two isolated clusters aopear to exist, one 
in the top portion of the plot and a second in the bottom pov^tion. It 
was felt that further investigation was warranted since the relationship 
of the total satisfaction scores to retrieval precision for the subjects 
isolated in the lower cluster of Figure 6 were highly correlated. 

It is noteworthy that all of the subjects contained in cluster 2 
obtain a search precision of zero with the exception of subject number 18 
who obtained a precision of 100. There is evidence that this subject's 
satisfaction was low because, even though he obtained maximal precision 
on his search, he received only nine hits but was expecting over 50. From 
this, we conclude that the cluster of subjects providing a lower total 
satisfaction score were those who became disenchanted with the experimental 
treatment because they received less than expected results from their 
information retrieval effort, thus influencing the linear regression analysis 
shown to be significant in Figure 5. There is no reason to believe that this 
is the fault of the interface" (EQUIP) . 

(See Figure 6 next page) 

Basically the question that was asked was the effect of this small 
cluster of subjects obtaining low precision for their search efforts responsible 
for a significant negative slope of the Bi line of the regression model? 
To investigate this possibility further, the two groups of subjects in 
question were separated and the linear analysis technique previously 
described was applied to each to determine if any regression effects 
existed for the isolated cluster. The results of these computations are 
provided in Figure 7. 

Figure 7 

Contrasted Linear Regression Sunmaary Calculation 
for Cluster 1 and Cluster 2 Subjects 





Cluster 


Dependent 
Variable 


Independent 
Variable 


(Intercept) 


F -value 
(Slope) (Regression) 


1 

(Top) 
N=21 


Total 

Satisfaction 
Score 


Gain 
Score 


15.8401 


0.364 1.75801 


2 

(Bottom) 
N=7 


Total Gain 

Satisfaction 

Score 


9,636 


0.0124 .0460 
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Critical F value for cluster 1 « F^95(l,21) = 4,30 
Critical F value for cluster 2 « F*oc(l,7) = 5.59 
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Supported by the analysis, it is concluded that the existence of a 
second different sample group within the experimental samole (cluster 2, 
Figure 6) is the cause of a significant regression effect at a = ,05 
when the entire experimental sample*s dependent total satisfaction scores 
are predicted using a linear model. However, because the R2 value of 19.7 
indicates that the model accounts for only approximately 20 percent of 
the variability of the total sample responses, and because two clusters, 
both lacking any significant regression effect were shown to exist, 
evidence suggests that CAI knowledge gained does not significantly affect 
user's satisfaction with his search effort. 

Additional Findings 

In addition to the observational data applied to each of the experimental 
hypotheses previously discussed, careful records of the retrieval precision 
of each subject were maintained. The precision of a search effort is 
expressed as a ratio converted to a percentage of the number of retrieved 
hits to the total number of citations retrieved by the ERIC/ QUERY system. 

It was felt that analysis of user's precision values was worthwhile. 
More specifically, does the retrieval precision of an expert (the information 
specialist) differ from that of a novice (the experimental treatment sample)? 
More formally, this question may be stated as a statistical hypothesis: 
There are no differences between the average precision rates of users 
undergoing CAI training and submission via EQUIP and those obtaining the 
expert services of an information specialist. 

To test this hypothesis. Student's t_ was applied, giving the results 
presented below. 



Figure 8 

Summary of Statistical Computation for Determining the 
Significance of Experimental Group Precision Means 
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F -ratio 


58.2 


52.679 


5.521 


.554 


.667 

• 



Note. --The F , ratio does not exceed F --(28,30) = 1.87 and 
^bs <'o«s not exciel t q7c(56) = 2.04. '^^ 
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Based on the non-significant value of t^bs; it is concluded that 
Ho:yt/^= may not be rejected. It makes no difference whether a search 
is encoded ^by an expert or a novice trained on a CAI interface program; 
ERIC/QUERY retrieval precision is uniformly low about 55 percent. 

INTERPRETATIONS OF EXPERIMENTAL RESULTS 

Among treatments 

To answer the pivotal issue of whether or not a specially constructed 
CAI interface program is viable, statistical tests were conducted to determine 
if population differences measured by the users' satisfaction responses ex- 
isted. Each of these tests was described. 

Statistical results of the testing procedure clearly indicated that 
the two interface methodologies did not cause any measurable differences 
in user's total satisfaction with the results of his search efforts. Analysis 
of individual items of the satisfaction questionnaire showed that, of the 
five, only one relating to the amount of time spent obtaining a search, 
measured any differences between the treatment populations. The experi- 
mental group was slightly less satisfied with the approximately two and 
one-half hours of time spent using the terminal as compared with control 
group members which spent less than thirty minutes in the search negotiation 
interview. 

Within the experimental treatment 

Examination of pre- and posttest gains of CAI interface users indicates 
that EQUIP did cause a significant increase in the novice user's ERIC/QUERY 
computer-searching skills. Surprisingly, however, there is no predictive 
relationship between such gains and the precision of the resulting searches 
when tested through linear regression techniques. Such a lack of predictability, 
it was felt, was due to the fact that precision attributes of ERIC/QUERY were 
such that, regardless of the computer-searching skill needed to execute a 
viable search strategy, precision rates remained l^ss than ideal. To test 
this possibility, a separate analysis was conducted to determine if the 
search precision of assumed experts was equal to that of novice users trained 
on the telecommunication terminal. No difference between mean search pre- 
cision of the two groups could be detected. This phenomenon may be due to 
any of three possibilities: (a) the novice users were as skilled as the 
information specialist; (b) regardless of skill, ERIC/QUERY is uniformly 
inefficient; or (c) a combination of both. Unfortunately, no empirical 
evidence may be brought forth which rectifies this issue; however, the 
results of the precision test between groups supports the use of EQUIP. 

In addition to conducting linear regression analysis to determine 
possible predictive relationships between EQUIP gain scores and user 
satisfaction, separate regression analysis was conducted for two cluster 
groups shown to exist within the sample — those subjects with very low 
retrieval rates (cluster 2) and all others (cluster 1). Due to the existence 
of the second cluster, it is evident that total satisfaction scores, and 
thus the significance of total subsample regression line, is effected, 
since separate regression analysis for each cluster was determined to be 
nonsignificant. This clearly shows the effect of some users' resulting search 
precision on their overall satisfaction of search effort an understandable 
situation since users do want access to information. 
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CONCLUSIONS 



Based on the procedures and analyses of collected data within the 
scope of the study, the following specific conclusions were formulated. 
These generalizations are confined to the specific population used in the 
study. 

The CAI Interface 

It is not only possible to use a CAI-medium tele-processing interface 
with which to bring novice users into direct contact with a complex 
computerized information retrieval system, but such an interface method 
(EQUIP) was created and demonstrated to function just as well as the search 
negotiations method in use at many installations. It was al^o demonstrated 
that populations of users classified as researchers or practitioners, based 
on pre-search retrieval expectations of the system, are both accommodated 
with the CAI interface in terms of their satisfaction with literature- 
searching efforts. Thus, the CAI interface services both types of users 
equally well . 

The Effectiveness of EQUIP 

Although EQUIP is one of many possible approaches to training users to 
submit a computer literature search, the significant pre- and posttest means 
coupled with the fact that all of the subjects in the experimental group 
were able to submit a search which was not rejected by the system clearly 
demonstrates the viability of such an interface method. It is concluded that 
EQUIP, as an interface process, did meet the objective for which it was 
created. 

General 

Tests of two of the three main study hypotheses designed to examine 
relationship between the experimental and treatment population resulted 
in the inability to reject the null hypotheses; i.e., no differences were 
found between the experiment groups within the dimensions studied. It is 
concluded that the CAI interface method, when used in conjunction with 
ERIC/QUERY, was no more nor less effective than search negotiations 
interface (which requires the use of "data processing types") even though 
knowledge gains were shown to exist among EQUIP users. 

IMPLICATIONS 

The Use of EQUIP 

For the past two years the user desiring ERIC/QUERY retrieval services 
had as his only alternative for initiating a search, the services of an 
information specialist a situation which is manifested in most 
computerized information retrieval systems desiring to continue receiving 
the advantages of batch processing. With the application of a CAI 
training and search submission interface, a new alternative has been 
demonstrated. The time utilized by an information specialist to initiate 
searches for users may confidently be relegated to the user and the computer. 
The number of users accommodated may be increased in direct proportion to 
the number of accessible telecommunication terminals available; thus a 
larger number of information seekers may gain access to information retrieval 
files at a time when the information need is critical. 
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It should be noted that the use of EQUIP, or similar CAI training and 
submission programs, is not limited to the ERIC/ QUERY system. Interface 
systf/ns such as EQUIP may be used with any_ batch processing system if mini- 
mum hardware and software requirements av^e provided. 

RECOMMENDATIONS 

The implementation of the concept of using a pre-dedicated, interactive 
systems mode to bring the educator (or anyone el5;e for that matter!) directly, 
into contact with the computer and, at the same time, not directly relying 
on the system's personnel time and services, may be considered a progressive 
step towards "humanizing" the data processing environment. Much has been 
done by many, as on-line systems have emerged. However, there is one very 
important ingredient that tends to make the interface system more 
attractive cost. 

It is felt that the alternative presented in this peper will become 
very attractive once cost comparisons between the EQUIP-type interface 
systems are compared and contrasted with existing on-line dedicated 
systems. It is emphasized that no empirical cost comparisons were attempt- 
ed In this study. This appears to be the next logical step, although the 
alternative offered here seems to be approximately 90% less costly than, 
for example, DIALOG (an on-line retrieval system which accesses the same 
ERIC files as EQUIP.) 

Finally, many educators are becoming increasingly concerned about 
claims by data processing personnel that the computer can be used to 
accomplish the tasks at the same costs as "old traditional" methodology. 
Educators have been heard to retort: "...but we aren't happy with what 
we are doing now. Give us alternatives which do things better !" If 
EQUIP-type interface systems are to be judged in light of the above, 
educators must, at least, attempt their use. 
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